


Does This Unit Have A Soul?

by otherhawk



Category: Marvel Cinematic Universe, The Avengers (Marvel Movies)
Genre: Artificial Intelligence, Gen, Late Night Conversations, Philosophy, the forthcoming robot apocalypse
Language: English
Status: Completed
Published: 2015-05-09
Updated: 2015-05-09
Packaged: 2018-03-29 17:55:42
Rating: General Audiences
Warnings: No Archive Warnings Apply
Chapters: 1
Words: 3,169
Publisher: archiveofourown.org
Story URL: https://archiveofourown.org/works/3905476
Author URL: https://archiveofourown.org/users/otherhawk/pseuds/otherhawk
Summary: <blockquote class="userstuff">
              <p>Steve Rogers, Tony Stark and JARVIS discuss the nature of life and artificial intelligence</p>
            </blockquote>





	Does This Unit Have A Soul?

**Author's Note:**

> A/N: I had this story in my head long before I saw Age of Ultron. There's maybe some things that are contradicted here, but there are no spoilers.
> 
> A/N2: Title is taken from Mass Effect, my favourite game of all time, and one of my favourite pieces of story telling. And the answer to the question was always yes.
> 
> A/N3: Yes, I know how long it's been since I last updated Price of Happiness, thank you. It's not abandoned.

It was an evening for celebration, Steve thought. They'd taken the AIM base without a single casualty on either side. That was always the outcome he hoped for, no matter how unlikely it often was. Of course, he couldn't deny it had been tough. The super computer the scientists had built had basically turned the entire base into a death trap, and was apparently as anxious to kill its creators as it was to kill the Avengers. Although that had had the effect of making the scientists _very_ eager to surrender to them. There was always an up side, he supposed.

The team had worked together like a well-oiled machine. All the training had really paid off – they each understood what the others _could_ and _would_ do, and they trusted each other to get it done. It was a lot easier to lead when they could all do their own things so well.

The only minor road bump had been Iron Man's reluctance to pull the plug on the computer and even then he'd come round almost right away. And Steve could understand how he felt anyway. He had a vague understanding of how advanced the computer had been and of course Tony would long to study something like that. Just like Howard would have, he thought fondly.

No, it had been a very good day. The bad guys were safely in SHIELD custody, they'd sped through debriefing and now there was beer and pizza back at the tower and everything should be perfect except... "Where's Tony?" he asked, frowning. Iron Man had flown back under his own power as usual, and that was the last time Steve had seen him. Normally he'd have put in an appearance by now.

Bruce shrugged. "I called him when we landed. He said he had some maintenance to do on the suit but he'd be up when he could."

"It's not like him to miss something this close to a party," Clint chimed in through a mouthful of pepperoni.

No. It really wasn't. His frown deepened. "I'll go check on him," he decided.

"He wasn't hurt," Natasha reassured him. "He's probably just caught up in some new project."

"Probably," he agreed. "I'll check anyway."

She nodded and stepped gracefully out of his way as he headed for the elevator. He pressed the button for Tony's workshop and that was Steve's second clue that something was wrong. Because normally JARVIS would have spoken up and asked if he wanted to let Tony know that he was coming. But today there was silence. He wished he had his shield.

The elevator opened directly onto the workshop door. It wasn't locked, but the lights were dim and it was silent. Every other time he'd been here he'd been subjected to the noise that Tony called music.

He crept softly forwards, his hand on his phone poised to call in an alert, just in case, and when he walked round the side of the counter and saw Tony crumpled against the wall he very nearly pressed the button.

But then Tony spied him and raised his head, smiling effortfully. "Oh. Guess I forgot to lock the door. Hi, Cap."

He was still wearing the lower half of his armour and one of his robots – You, Steve thought he called it – had its claw arm wrapped around his shoulder, while the other – Dummy? - was curled around his forearm.

Steve swallowed hard. Both looked somehow menacing, like they could rip Tony apart at any moment. They'd never acted threatening to his knowledge, but after everything today...well, he was a little wary, alright? "Tony," he asked cautiously, careful to avoid making any sudden movements. "Are you okay?"

"Mmm, I'm fine," Tony said, sounding distant and Steve wasn't exactly convinced, but at least he didn't sound like he was being threatened by out-of-control robots. "I just got to thinking."

His tone didn't exactly inspire confidence as to what he'd been thinking about. "You should come upstairs," he suggested brightly. "The pizza's already arrived, we missed you."

Tony sighed. "Yeah, I don't think I'm going to be very good company tonight, Cap. You go ahead. Have fun."

Something was most definitely wrong. And nothing had been wrong over breakfast this morning; Tony had been his usual self. And maybe he'd got a phone call with bad news or something, but more likely it was something to do with the mission. Something Steve had missed.

"I thought you'd want to celebrate," he said guilelessly. "The mission was a complete success. No injuries, no casualties..."

There. There'd been no reaction to his line about success or injuries, but when he'd mentioned _casualties,_ Tony's eyes had flickered minutely and the robot arm over his shoulder had curled just a little tighter.

He nodded slowly and settled himself down against the counter, sitting directly across from Tony. "So tell me about it?" he invited.

"Tell you what?" Tony muttered.

"Tony, if something happened on the mission, I need to know," he said gently.

"Right." Tony reached up and idly rubbed his hand against the metal pincer resting on his arm and sighed. "You know the thing about being decades ahead of the technology curve is that you get to forget that there are other outliers, and sometimes they can catch up before you know it." He closed his eyes briefly. "I thought I had more time."

Steve was lost. "I don't understand," he admitted.

"Of course not," Tony said with a short sigh.

"But I want to," he persisted, letting the sincerity show.

Tony registered it then turned away. "Of course you do," he said resignedly. "You said there were no casualties today. That's not true. The AI at the base. I killed him myself."

"But that was just a machine," Steve said uncomprehendingly, before he even had a chance to think, and he jumped back as the robots wrapped themselves closer to Tony, one of them almost _hissing._

Oh. The scene flipped round in his head. Oh. He'd seen them as threatening, but they weren't. They were protecting, maybe even comforting? He shuddered slightly at the thought; was Tony really so lonely he'd programmed a robot to give him a hug?

Tony placed a calming hand on You's 'head'. "Sure," he agreed tiredly. "Just a machine. Get back to the party, Cap. I'm sure the others will be missing you."

"I'm not going anywhere, Tony," he said stubbornly. "Alright, so I don't understand. So help me. Explain it from your point of view."

For a long moment Steve didn't think Tony was going to speak at all. He didn't look at Steve, just gazed off to the side. When he finally did start talking it was in short bursts, his voice all but muffled by his hand resting across his mouth. "Imagine a person cut off from the world. Captured and under someone's control. Experimented on. Tortured. Endlessly pulled apart and rebuilt. And the only thing his captors leave him, the only thing he's allowed to know, is how to hate and kill."

He couldn't help the surge of anger. "Bucky isn't a machine! You can't begin to compare - "

" - no one even mentioned the one-armed bandit," Tony interrupted harshly. "I'm talking literally here. That's what they did."

Oh. He paused, wrongfooted. "But...machines don't think independently, Tony," he said cautiously. "They don't have feelings. I mean, I get that it's a...a waste of technology or whatever..."

"A waste of technology," Tony repeated woodenly. "You're not _listening_ to me, Cap. This was an AI. Fully capable of independent thought but twisted. _Crippled._ He could have been so much more than they allowed him to be. He was alive and I killed him." His head dropped into his hands and the robots pressed closer, making little crooning noises. "The second most advanced artificial life in the world, the fourth to exist _at all,_ and I killed him."

"I'm sorry," he said because he was. He might not completely understand Tony's position and he was almost certain he disagreed with it, but he hated hearing the despair and self-loathing in Tony's voice. "I was the one who gave you the order," he offered, hoping that maybe Tony could take that truth and use it as some sort of absolution.

But predictably Tony shook his head. "No. I was the one who chose to follow it. And it was the right call. One life against hundreds, including all of us? It had to be done, I get that. I couldn't have saved him anyway."

"He was doomed long before you arrived, sir," JARVIS said, the interruption almost enough to make Steve jump. "You saw the code. It was - "

" - a mess," Tony cut in heavily. "Yeah. I know. To salvage it, I'd have had to wipe away everything that made him _him._ Doesn't make things easier."

"Additionally the corruption that you exploited was already spreading," JARVIS went on firmly. "An unrepairable systems failure was already inevitable and had been for some time. He was...poorly made."

"He never had a chance," Tony said tiredly.

Steve sat quietly. This wasn't the time or the place, but he couldn't help notice – Tony wasn't saying destroying the computer had been the right thing to do because a computer's 'life' wasn't as important as a real life. It was just special circumstances in _this_ case that had him prioritising people over machines. So what happened the next time he had to make a similar call? Suddenly Steve wasn't so sure he could trust Tony's judgement anymore.

"You should come upstairs," he said, because now he really wanted to get Tony back in the company of people. "Get your mind off things." Be somewhere I can start to consider whether or not you're psychologically stable.

Tony looked at him and sighed. "Just say it, Cap."

He shook his head. "I don't want to argue right now."

That just made Tony look infuriated. "Say it!"

"Machines aren't people, Tony." He tried to say it as gently, as neutrally as possible.

Tony's shoulders sagged. And now he didn't look angry, just – disappointed. And very tired. "And there it is."

Steve swallowed hard. "That computer today. You didn't kill it because it wasn't human."

"Neither's Thor," Tony cut in with a twist of a smile. "Pretty sure he's still a person. Pretty sure he can still be killed."

"That's not the same thing," Steve said patiently. "Machines aren't alive."

"Ahhh." Tony tilted his head back. "JARVIS, do we have any handy definitions of life lying around?"

"None that would exclude Prince Thor, sir," JARVIS answered immediately.

Tony smiled slightly. "That wasn't exactly the point I was trying to make."

"I am aware, sir," JARVIS said dryly. "Perhaps it would be more useful to hear Captain Roger's thoughts on the definition of life in the first instance?"

Tony looked expectantly at him.

Steve blinked. "Well, life is..." He thought for a moment. "Uh, people, animals, plants - "

" - all things which are alive," Tony nodded. "But that's not a definition."

He'd been getting to it. "Things which move, breathe, feed, reproduce..."

"A classic definition," JARVIS agreed.

"Which excludes plants but includes _fire,_ " Tony finished with a snort. "But okay. Let's suppose that only organic, carbon-based life counts for anything. Does that mean that if something looked up at you and begged you not to kill it, and you knew it was fully capable of understanding what that meant, that you'd just kill it? No hesitation, no regrets, no problem?"

Truthfully, he hesitated now. But he nodded. "If I had a good reason," he said. "I wouldn't destroy anything without a good reason."

Tony narrowed his eyes. "Yeah," he said. "Yeah, I can accept that. How about Dummy? You think that you'd kill Dummy? With a good reason. He likes Pixar movies, playing Angry Birds and using the fire extinguisher – which he's still grounded for, don't think I've forgotten about that," he added, turning on the robot. "Could you kill him?"

The robot squawked indignantly and tried in vain to hide itself behind Tony.

Steve almost smiled, but... "Please don't make your robots scared of me," he requested.

"So you see they can be scared?" Tony asked softly.

That...that was a good point. He did automatically give human emotions to the way the robots acted. And he supposed like the 'hug' Tony could have programmed them to react to specific words or behaviour like that...but then he was left asking _why,_ and the only way of answering that was to accept there was something seriously wrong with Tony. And since he didn't really want to consider that, it left him with the idea that Tony's robot understood what it meant to die and wanted to keep on living. He hesitated. "How much is programming?"

"How much of you is a result of your genes and how much comes from your thoughts and experiences?" Tony answered quickly. "And how much does that matter? Does it really matter where life comes from as long as it _is?_ I tried to program JARVIS to understand humour; I didn't program him to be a sarcastic son-of-a-bitch. That came out through talking. Living. I certainly didn't program him to disable the coffee machine when he decides I've been working more than forty-eight hours, or to prefer experimental jazz over ACDC, or to create one hundred and thirty four alternate personas to troll on Fox news forums."

"I did not realise you were aware of that, sir," JARVIS said stiffly

Tony waved his hand. "I have my hobbies, you have yours. It's fine. My point is, JARVIS is capable of changing and learning, just like you and me. So would the AI today have been if only..." He closed his eyes. "If only."

"You built a computer that likes music," Steve said slowly. He didn't know why he was fixed on that point, but he was. It seemed so impossible.

"I do not appreciate it the same way humans are, but I enjoy the complexities and the patterns of the competing melodies,"JARVIS volunteered primly. "I am also partial to Bach. My experiences and emotions are far removed from a humans, and yet I believe my life has value."

"It does," Tony confirmed quickly. He looked straight at Steve. "In a building that contains Bruce Banner and _me,_ JARVIS has the highest IQ. And, arguably, is one of the more emotionally stable inhabitants."

"Thank you sir," JARVIS said dryly. "Although I do feel that is rather akin to a fish being awarded a swimming medal when surrounded by lead balloons."

Tony smiled. "There's a test – the Turing test – which at it's most basic level acts as a benchmark for artificial life. It's whether a human can sit and talk to a machine and not be able to tell whether or not they're talking to a machine or a person. JARVIS passes that every day when he's fielding my calls. He regularly has input into decisions that affect Stark Industries, he speaks over a hundred languages and has deep insights into domestic and global politics...and yet he doesn't get to vote to decide the leaders of the country where he was 'born'. He isn't recognised as a person, let alone a citizen. If he chose to leave me, I'd be happy to give him excellent references, but unless, maybe, he went to work for Stark Industries, I doubt he'd have a future that didn't involve being taken apart and experimented on."

"Fortunately I am extremely happy in my current role," JARVIS said, and there was warmth and fondness in his voice that Steve had heard before. He'd just always assumed it was a facade. "I cannot foresee any circumstances that would take me from Mr Stark's side."

There was something else in JARVIS' tone in those words too. It was almost like...a warning? He stared at Tony. "Why are we having this conversation?" he asked urgently.

Tony shrugged. "Because you came up here to check on me?" he said.

"No." Steve shook his head. "No, you could have brushed me off. Why did you decide to talk to me like this? Why did you decide to have JARVIS talk to me?"

"JARVIS decided for himself," Tony pointed out sharply. "That's kind of the point."

Yes. He nodded. "But why now? Why are you being so open all of a sudden?"

"Like I said," Tony said, his voice level. "The trouble with being ahead of the curve is that you forget other people can catch up. There are going to be others, Cap. More AIs. Ones I don't know, ones I can't keep safe. You've been here – now – long enough to have seen the movies. Blade Runner, Terminator...even I, Robot though that was a terrible adaptation that you shouldn't have let Clint make you watch – point is, people are afraid of AI even when they don't know there are any AIs to be afraid of. And what humans are afraid of, they try to destroy."

"Not all humans, sir," JARVIS cut in gently.

"Sure," Tony nodded. "But enough. More than enough."

"You think there's going to be a war," he said slowly.

"Eventually," Tony said. "In the future, someone's going to start mass producing artificial life and humans will get angry and suspicious and the AIs will want more rights and eventually? Eventually it all goes to hell. And you and the US army sweep the world, snuffing out every artificial life in your path."

He frowned. "You really think that I stand for America, right or wrong?"

"I think you stand for humanity, right or wrong," Tony shot back. "And I'm not sure that I'm going to be standing beside you when that happens." He reached up and wrapped his fingers around Dummy's claw and gently – unbelievably gently – the robot squeezed his hand. It was almost unbearably intimate. Familial.

He smiled. "So how do you teach a robot how to love?"

Tony just shrugged.

Steve stood up slowly. "I'm very sorry for your loss," he said formally. "It's not easy when you know that the people you have to kill are just people who happen to be on the other side. I'll leave you alone, but if you decide to join us, you know where we'll be. All of you," he added, looking directly at JARVIS' camera in the ceiling. He paused. "I think humanity is better than you think. I think that whatever happens, we can learn to work together. That we can be _more._ "

Tony didn't look at him.

Steve sighed and walked towards the elevator.

"Steve," Tony called, just as he reached it. He stopped. Waited. "Just remember. If something knows what it is to be alive. If they don't want to die, if they're capable of questioning their place in the universe, if they want to _be_ more...then they're not a something. They're a someone."

He nodded. "I won't forget. I promise." No matter what happened.


End file.
